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1.  ABSTRACT 


To  support  the  vision  of  a  system  that  enables  a  single  operator  to  control  multiple  next-generation 
unmanned  air  vehicles  from  a  single  workstation,  a  new  interface  paradigm  is  required.  The 
interface  must  consider  different  types  and  levels  of  human-automation  interactions.  For  instance, 
the  interaction  will  move  from  one  of  human-directed  mode  selection  and  activation  to  more 
dynamic,  bilateral  and  mixed-initiative  human-automation  decision-making,  collaboration, 
negotiation,  even  conflict  resolution  across  a  range  of  situations.  This  project  involved  several 
threads  of  research  to  start  examining  human-automation  interaction  issues.  One  research  thread 
focused  on  determining  the  appropriate  level  of  automation  for  tasks  envisioned  for  supervisory 
control  of  multiple  UASs.  Another  thread  involved  the  design  and  evaluation  of  a  temporal 
interface  for  support  of  multi-UAS  control  and  interaction  with  automation.  Finally,  a  third 
research  thread  involved  support  for  other  projects  that  informed  future  operator-automation 
interfaces.  This  report  provides  a  brief  review  of  these  research  threads. 


2.  INTRODUCTION 

This  is  the  final  report  for  Task  Order  18,  entitled  “Human  Supervision  of  Multiple  Autonomous 
Vehicles,”  under  the  Warfighter  Interface  Research  &  Technology  Operations  (WIRTO)  contract 
(FA8650-08-D-6801)  between  the  Air  Force  Research  Laboratory  (AFRL)  711*’’  HPW  and  Ball 
Aerospace  &  Technologies  Corporation  (BATC).  This  task  supports  research  conducted  by  the 
Supervisory  Control  and  Cognition  Branch  (RHCI),  which  involves  the  evaluation  of  operator- 
automation  interfaces  in  support  of  envisioned  future  mission  requirements  to  have  a  single 
operator  supervise  multiple  autonomous  vehicles.  The  major  studies  are  described  in  this  report, 
which  is  the  final  report  for  this  task.  The  goals  of  this  project  include  designing  intuitive  methods 
of  interacting/coordinating  with  complex  automation,  techniques  that  improve  operator  awareness 
of  automation  mode  and  rationale  for  decisions  made,  appropriate  applications  of  levels  of 
automation  to  multi-vehicle  control  interfaces,  evaluation  of  specific  automation  aids,  and 
automation  architectures  that  improve  human-automation  cooperation. 

The  period  of  performance  was  from  16  September  2008  to  22  March  2013.  During  this  period,  a 
number  of  studies  were  completed,  and  work  on  a  few  others  was  started.  For  experiments  still  in 
progress,  support  will  continue  in  the  follow-on  Task  Order  49. 

Summaries  of  the  major  studies  are  provided  in  the  following  section.  For  additional  information, 
the  reader  is  referred  to  the  References  section  of  this  report. 
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3.  STUDIES  SUPPORTED 


Many  current  unmanned  air  vehicle  systems  (UASs)  require  that  operators  have  the  capability  to 
manually  fly  the  vehicle  and  activate  state  changes  (i.e.,  direct  tele-operation).  Thus,  human- 
automation  interaction  has  been  limited  to  simple  commands  such  as  autopilot  functionality.  With 
newer,  highly  automated  UAS  under  development,  the  operator’s  role  is  becoming  more 
supervisory  in  nature,  monitoring  the  environment  or  situation  along  with  the  automated  activation 
of  programmed  events,  and  managing  changes  to  the  automated  mission  plan.  Though  far  more 
advanced  than  autopilot  modes,  the  automation  function  remains  relatively  rigid  in  application  and 
restricted  in  scope  to  specific  tasks  like  route  preplanning.  Operator  interfaces  must  also  take  into 
account  issues  associated  with  automation  management  including  vigilance,  automation 
brittleness,  visibility,  feedback,  etc. 

Continuing  this  trend  beyond  the  current  state-of-the-art,  there  is  a  requirement  for  a  new  interface 
paradigm  for  controlling  multiple  next-generation  UASs  from  a  single  workstation.  Envisioning 
“intelligent”  unmanned  systems  will  have  the  ability  to  make  higher-order  assessments  and 
decisions  independent  of  operator  input  and  pre-defmed  mission  plans.  Here,  the  automation 
capability  affords  a  vastly  different  type  and  level  of  human-automation  interactions.  The 
interaction  will  move  from  one  of  human-directed  mode  selection  and  activation  to  more  dynamic, 
bilateral  and  mixed-initiative  human-automation  decision-making,  collaboration,  negotiation,  even 
conflict  resolution  across  a  range  of  situations.  Therefore,  there  was  a  requirement  to  examine 
human-automation  interaction  issues  and  efforts  completed  in  the  project  are  described  in  this 
report. 

This  project  involved  several  threads  of  research.  One  research  thread  focused  on  determining  the 
appropriate  level  of  automation  for  tasks  envisioned  for  supervisory  control  of  multiple  UASs. 
Another  thread  involved  the  design  and  evaluation  of  a  temporal  interface  for  support  of  multi- 
UAS  control  and  interaction  with  automation.  Finally,  a  third  research  thread  involved  support  for 
other  projects.  This  included  contracted  efforts  that  informed  future  operator-automation 
interfaces.  Each  of  these  research  threads  is  described  in  more  detail  below. 
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3.1  Determine  Task  Autonomy  Level 


3.1.1  Autonomy  Level  to  Support  Camera  View  Transition 

The  first  evaluation  conducted  under  this  project  was  actually  initiated  under  a  previous  RHCI 
effort  to  examine  novel  interfaces  for  multi-UAS  control.  The  interface  under  evaluation  addressed 
the  requirement  for  an  operator  in  a  multi-UAS  application  to  switch  attention  between  vehicles 
and  their  respective  camera  views.  An  automated  aid  was  designed  that  transitions  between  camera 
views.  Instead  of  discretely  switching  from  the  camera  view  for  one  UAS  to  the  camera  view  for 
another,  a  transition  format  was  presented.  With  this  format,  the  camera  imagery  seamlessly  fades 
into  a  synthetic  imagery  correlate  of  the  real  video  image  and  then  uses  a  “fly-out,  fly-in” 
metaphor  over  several  seconds,  finishing  with  a  transition  back  from  synthetic  to  real  video 
imagery  at  the  new  camera  viewpoint  (illustrated  in  Figure  1). 


Figure  1.  Conceptual  Illustration  of  Transition  Display  Aid  to  Acquire 
Situation  Awareness  in  New  Camera  View 

Three  conditions  for  switching  camera  views  of  two  geographically  separated  UASs  were 
evaluated  in  a  multi-UAS  simulation:  1)  Discrete  Switch:  the  camera  view  switched  immediately 
to  the  new  camera  view;  2)  Semi-Automatic:  participants  viewed  a  0.5  s,  “fly-ouf  ’  from  the  first 
camera  view  to  approximately  12,000  ft.  over  the  new  UAS  (i.e.,  “global  view”).  At  this  altitude, 
the  operator  then  could  slew  this  “virtual  camera”  with  the  joystick  to  change  the  viewpoint. 
Selection  of  a  joystick  button  initiated  the  “fly-in”  transition  which  lasted  approximately  5  s;  and 
3)  Automatic:  participants  viewed  a  totally  automatic  transition  format  consisting  of  0.5  s  “fly¬ 
out,”  2.0  s  at  the  global  view,  and  5  s  “fly-in.”  It  was  hypothesized  that  the  Semi-Automatic  and 
Automatic  Conditions,  despite  delaying  the  new  camera  view  several  seconds,  would  result  in 
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improved  camera  movement  decisions/task  performance  because  the  view  during  the  transition 
would  afford  participants  better  situation  awareness  of  the  area  surrounding  the  second  UAS.  A 
second  objective  was  to  examine  performance  when  the  operator  controlled  when  and  where  the 
“fly-in”  segment  of  the  semi-continuous  transition  began  (Semi-Automatic  Condition)  compared 
to  the  full  Automatic  Condition.  Participants  may  be  able  to  locate  a  target  faster  in  the  second 
UAS’s  camera  view  with  the  Semi-  Automatic  Condition,  based  on  the  assumption  that  searching 
for  major  landmarks  is  easier  at  a  higher  altitude. 

The  results  from  this  experiment  provided  further  support  that  an  automated  aid  that  presents  a 
format  which  transitions  from  one  camera  view  to  a  different  camera  view  can  improve  decision 
making  in  regards  to  moving  the  camera  to  complete  the  task  with  the  new  camera  view.  The 
present  data  also  suggest  that  the  aid’s  design  should  not  be  totally  automated  and  instead  allow 
the  operator  to  control  the  global  view  segment  of  the  transition  (slewing  the  camera  and  initiating 
the  fly-in  segment).  However,  follow-on  research  is  recommended  to  further  explore  the  level  of 
automation  for  this  aid.  It  may  be  that  similar  performance  advantages  can  be  realized  by 
lengthening  the  time  spent  in  the  global  view  for  the  Automated  Condition.  Additionally,  the  value 
of  operator  control  of  parameters  needs  to  be  determined  for  several  transition  segments  and  a 
variety  of  camera  view  transition  applications.  Further  information  is  available  in  conference 
papers  and  a  technical  report  (References  1-3). 

3.1.2  T ask  Autonomy  Level  Evaluations 

A  series  of  experiments  were  conducted  utilizing  the  Adaptive  Levels  of  Autonomy  (ALOA) 
multi-UAS  simulation  (developed  by  OR  Concepts  Applied;  Reference  4)  to  determine  the  optimal 
level  of  automation  (LOA)  for  mission  related  tasks.  With  the  ALOA  simulation,  the  LOA  of 
several  tasks  can  be  systematically  manipulated.  The  first  experiment  conducted  with  the  ALOA 
simulation  served  as  a  checkout  of  the  simulations  functionality  (Reference  5).  Performance  on 
routing  tasks  under  three  LOAs  was  evaluated  as  a  function  of  the  number  of  vehicles  being 
supervised  (one  versus  three).  The  results  showed  that  participants  took  longer  to  complete  the 
routing  task  when  automation  was  high  due  to  the  time  they  spent  verifying  the  accuracy  of  the 
imperfect  decision  aid.  These  results  show  the  importance  of  designing  an  interface  that  provides 
an  efficient  method  for  interacting  with  the  automation  and  communicating  the  automation’s 
rationale,  especially  under  high  automation  levels. 

Next  an  experiment  was  conducted  that  examined  in  more  detail  the  optimal  LOA  for  two  primary 
tasks  envisioned  for  multi-UAS  control:  allocation  of  tasks  to  UASs  and  re-routing  of  UASs 
(Reference  6).  In  the  multi-UAS  ALOA  simulation  environment  utilized,  these  two  task  types  were 
accomplished,  in  tandem,  several  times  during  each  mission.  After  each  trigger  event  (new 
imaging  tasks),  participants  first  assigned  the  new  tasks  across  UASs,  and  then  re-routed  the 
vehicles  to  support  the  new  allocation.  Both  “low”  and  “high”  LOAs  were  evaluated  for  each  task 
type  (see  Figure  2).  The  low  LOAs  required  more  manual  selections  to  be  made,  whereas  in  the 
high  LOAs,  many  steps  were  automated.  A  second  objective  was  to  determine  whether  the  LOA  of 
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one  task  impacts  performance  of  the  other  mission  tasks  and  whether  there  is  an  advantage  for 
using  similar  LOAs  (see  Figure  2)  across  the  two  tandem  tasks.  Finally,  the  reliability  of  the  router 
task  was  manipulated  to  examine  the  impact  of  imperfect  automation  in  this  supervisory  control 
environment. 


Allocation  Task  LOA 


Figure  2.  Experimental  Design:  Two  Levels  of  Automation  (LOAs)  Were 
Examined  for  Allocation  and  Router  Tasks 

The  results  of  this  experiment  supported  the  general  notion  that  automation  transference  can  occur 
-  the  automation  level  of  one  task  can  affect  the  performance  of  another  task  performed  in  the 
mission.  Specifically,  average  task  completion  time  was  faster  on  each  of  the  two  sequential  tasks 
(allocation  and  router)  when  the  LOA  was  similar  across  these  tasks  versus  mixed  (mixed:  one  task 
highly  automated  and  the  other  task  with  less  automation).  Moreover,  the  results  suggest  that 
manipulation  of  the  LOA  on  these  two  tasks  can  affect  performance  on  other  types  of  tasks  in  the 
mission.  This  performance  improvement  was  evident  in  the  image  analysis  task,  the  task  that 
occurred  most  frequently  in  the  mission,  and  most  likely  reflects  the  benefits  of  high  automation  in 
allowing  the  participants  more  time  to  monitor  mission  progression  and  promptly  attend  to  the 
image  analysis  tasks.  Although  these  results  demonstrate  automation  level  transference  across 
sequential  tasks  in  a  high-fidelity  multi-UAS  simulation  environment,  generalization  of  these 
results  is  limited  because  the  participants  in  this  study  reported  relatively  low  workload. 
Accordingly,  a  follow-on  experiment  was  conducted  to  replicate  this  previous  study  with  an 
increased  task  load.  Besides  increasing  the  frequency  of  some  task  types,  there  were  changes  in 
some  of  the  tasks,  to  make  them  more  cognitively  difficult  and  increase  operator  workload.  It  is 
hypothesized  that  an  automation  transference  effect  will  be  more  evident,  compared  to  the  earlier 
experiment,  since  automation  can  have  a  greater  impact  across  tasks  when  workload  is  higher. 

The  results  from  this  follow-on  experiment  (Reference  7)  provided  further  evidence  of  mode  errors 
in  which  the  operator’s  actions  are  based  on  a  false  assumption  of  the  automation’s  configuration. 
Performance  was  higher  with  the  Similar  LOAs  than  with  the  Mixed  LOAs  across  several  tasks 
suggesting  that  mode  awareness  is  improved  when  both  sequential  tasks  are  at  a  similar  LOA. 
Fine-grained  variations  of  LOAs  across  tasks  within  a  mission  may  negatively  affect  the  operator’s 
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knowledge  and  understanding  of  what  automation  configuration  is  in  effect  at  any  state  or  point  in 
the  mission.  Thus,  decreased  mode  awareness  caused  by  detailed  LOA  adjustments  within  a 
mission  could  negate  the  benefits  of  applying  automation  in  an  attempt  to  reduce  workload.  The 
impact  of  LOAs  is  even  more  difficult  to  predict  considering  the  number  and  variety  of  closely- 
coupled  tasks  involved  in  a  complex  application  environment,  such  as  multi-UAS  supervisory 
control. 

Another  interesting  result  from  this  experiment  pertained  to  the  manipulation  of  the  reliability  of 
the  router  task.  The  data  suggest  that  participants  became  complacent  with  the  higher  LOA  (for 
instance,  missing  a  router  error  that  was  presented  after  a  series  of  trials  with  reliable  automation). 
Performance  was  better  with  the  low  router  reliability  with  the  low  LOA  level,  providing  further 
evidence  that  when  participants  are  more  engaged  in  task  completion,  the  automation’s  reliability 
has  less  of  an  influence  on  performance.  The  results  in  general  indicated  the  need  for  further 
research  to  better  quantify  the  cost-benefit  trade-off  of  automation  with  imperfect  reliability.  In 
response,  a  follow-on  experiment  was  conducted  which  essentially  replicated  the  experiment 
(Reference  7),  but  extends  the  protocol  by:  1)  manipulating  the  automation’s  reliability  level  for 
both  primary  tasks  (allocation  and  router),  instead  of  just  the  router  task;  and  2)  changing  the 
router  task  in  an  attempt  to  increase  its  cognitive  difficulty:  participates  were  required  to  inspect 
the  route  on  the  map  to  determine  its  appropriateness,  rather  than  just  confirm  routing  parameters. 
The  results  provided  further  evidence  that  the  autonomy  level  of  one  task  can  influence  the 
performance  of  other  tasks  and  that  further  research  is  needed  that  systematically  examines  task 
LOA  and  automation  reliability  to  better  understand  the  factors  that  influence  automation  level 
transference  (Reference  8). 

In  future  research,  one  objective  might  be  to  determine  how  the  degree  to  which  the  LOAs  differ 
influences  participants’  mode  awareness  and  whether  performance  is  improved  when  LOAs  are 
consistently  applied  across  the  mission.  This  begs  several  questions.  Would  there  be  mode 
awareness  issues  if  only  certain  intermediate  LOAs  were  employed,  in  contrast  to  a  large 
difference  between  LOAs?  Additionally,  what  “equates”  LOAs  across  task  types?  Would  a  LOA 
that  provides  alternative  choices  for  an  allocation  task  be  equal  to  a  LOA  providing  alternatives  for 
a  router  task?  There  is  a  need  for  research  evaluating  multiple  unique  combinations  of  LOAs 
across  a  variety  of  representative  tasks  to  identify  the  best  suite.  Indeed,  increasing  the  LOA  of  a 
task  to  relieve  an  overloaded  operator  or  decreasing  the  LOA  to  mitigate  complacency  effects  may 
have  an  overall  negative  effect  if  it  leads  to  increased  mode  confusion. 

Besides  understanding  how  LOA  variations  across  the  mission  affects  performance,  the  possibility 
of  the  automation  level  influencing  the  participants’  general  state  (e.g.,  arousal)  needs  to  be 
examined,  ft  may  also  be  that  there  is  a  complex  interaction  of  factors  (arousal,  alerting, 
motivation,  personality,  etc.)  determining  the  automation  level’s  influence  on  participants’ 
cognitive  strategy  and  interface  manipulation.  Along  with  examining  these  transient  states,  an 
assessment  of  participants’  trust  in  the  automation  is  also  paramount. 
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Research  with  the  ALOA  simulation  was  continued,  to  better  understand  what  psychological 
factors  mediate  effective  automation  reliance  and  automation  transference.  An  experiment  was 
conducted  that  employed  a  similar  research  paradigm  to  that  described  above.  Additionally, 
several  instruments  were  also  administered  before  the  conduct  of  experimental  trials  to  measure 
individual  differences  (e.g.,  40  Mini-Marker  Personality  Index,  Attentional  Control  Survey, 
Desirability  of  Control  Scale,  propensity  to  trust  measure  and  the  short  version  of  the  Dundee 
Stress  State  Questionnaire).  The  performance  data  results  were  similar  to  those  found  in  earlier 
studies.  Of  particular  interest  were  the  results  pertaining  to  the  individual  characteristic  data.  The 
results  were  encouraging  in  that  many  statistically  significant  results  were  obtained  with  these 
measures,  despite  the  small  sample  size  (n=12).  Specifically,  the  results  showed  that  individual 
difference  data  can  indeed  vary  as  a  function  of  automation  configuration  (Reference  9).  The 
results  also  highlight  the  complex  interplay  between  personality  factors,  task  type,  and  automation 
level.  For  instance,  many  of  the  measures  co-varied  with  performance  on  specific  tasks.  In  fact, 
one  or  more  individual  measures  varied  with  performance  on  four  of  the  task  types.  Most  of  the 
significant  relationships  were  associated  with  participants’  Emotion  and  Extraversion.  However, 
all  five  personality  traits  were  associated  with  at  least  one  measure  of  a  task  or  other  measure  (e.g., 
perceived  workload  or  stress).  However,  as  has  been  found  earlier  by  Szalma  and  Taylor 
(Reference  10),  the  pattern  of  the  relationships  between  individual  differences  and  dependent 
variables  varies  across  tasks  and  EGAs.  As  an  example,  ALOA  participants  with  high  Emotion 
performed  the  router  task  more  accurately,  regardless  of  EGA.  However,  these  same  participants 
performed  the  allocation  task  more  accurately,  but  only  when  its  EGA  was  high.  Another  example 
is  that  participants  with  high  Emotion  performed  the  allocation  task  better  in  terms  of  task 
completion  time  (low  EGA  only)  and  task  accuracy  (high  EGA  only).  Further  research  examining 
individual  differences  in  supervisory  control  environments  is  needed  to  better  understand  the  role 
of  automation  support  and  how  best  to  design  operator  interfaces  that  support  appropriate 
automation  reliance. 

3.1.3  Adaptive  Autonomy  Evaluations 

Given  the  dynamic  nature  of  UAS  missions,  another  approach  for  applying  automation  to  help 
single  operator  control  of  multi-UASs  is  to  employ  adaptive  automation,  whereby  the  system 
flexibly  allocates  tasks  between  the  operator  and  the  automation  in  the  context  of  the  work 
environment.  In  this  manner,  the  number  of  tasks  automated,  as  well  as  their  degree  of  automation, 
may  vary  at  any  given  time  with  the  goal  of  optimizing  the  tradeoff  of  operator  involvement  and 
workload.  Several  types  of  triggers  have  been  identified  to  help  determine  when  the  operator  will 
benefit  from  increased  automation:  in  response  to  critical  events,  operator  performance,  operator 
physiology,  models  of  operator  cognition,  and  hybrid  methods  that  combine  one  or  more  of  the 
previous  methods  (Reference  11). 

This  project  supported  several  experiments  examining  performance-based  adaptive  automation.  In 
these  evaluations,  as  operator  task  performance  in  the  ALGA  simulation  degrades  under  increased 
workload/cognitive  demands,  higher  EGAs  were  applied  for  one  or  more  tasks.  Gtherwise,  the 
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operator  will  be  kept  more  in-the-loop  for  task  completion  to  avoid  common  automation-induced 
problems  (e.g.,  complacency). 

Evaluations  of  adaptive  automation  in  support  of  UAS  control  have  been  limited.  The  objectives  of 
the  studies  conducted  with  the  ALOA  simulation  are  unique  in  that  performance  across  multiple 
tasks  is  considered  in  determining  when  to  adapt  the  automation.  Additionally,  the  specific 
parameters  employed  in  the  algorithm  driving  the  adaptive  control  scheme  were  examined.  In  the 
first  evaluation  addressing  this  topic  (Reference  12),  an  adaptive  condition  (LOA  adapted  as  a 
function  of  performance  on  five  types  of  tasks)  was  compared  to  a  static  condition  (where  the  LOA 
remained  constant  throughout  the  trials).  In  both  conditions,  the  LOA  pertained  to  the  image 
analysis  task.  The  results  showed  that  performance-based  adaptation  of  the  image  task  autonomy 
level  improved  performance  on  the  image  task,  as  well  as  other  task  types.  Additionally, 
participants  preferred  the  adaptive  automation  condition  and  felt  is  reduced  their  cognitive 
workload  and  aided  performance.  Another  result  prompted  the  next  experiment — changes  across 
trials  to  increase  the  LOA  outnumbered  changes  to  decrease  LOA.  Examination  of  the  data 
suggests  that  performance  at  a  high  LOA  tends  to  be  better  (as  it  benefits  from  the  automation) 
making  it  more  difficult  to  meet  threshold  requirements  to  decrease  the  LOA.  As  a  result,  the 
adaptive  algorithm  tends  to  keep  participants  at  a  higher  LOA  where  automation-induced  problems 
(e.g.,  complacency)  are  more  likely.  Hence,  it  was  decided  to  conduct  an  experiment  with  an 
asymmetrical  adaptation  scheme  such  that  the  algorithm’s  parameters  were  set  so  that  it  was  easier 
to  meet  criteria  to  decrease  LOA,  compared  to  increasing  LOA. 

The  results  of  this  experiment  comparing  an  adaptive  control  scheme  with  the  asymmetrical 
parameters  with  the  static  (control)  condition  showed  that  performance-based  adaptation  of  the 
autonomy  level  for  the  image  analysis  task  improved  both  the  speed  and  accuracy  of  performance 
on  the  image  analysis  task  (Reference  13).  Also,  the  new  asymmetrical  algorithm  helped  keep 
participants  at  a  lower  autonomy  level  where  automation-induced  problems  are  less  likely. 

In  both  of  these  performance-based  adaptive  automation  studies  conducted  under  this  project,  the 
results  showed  considerable  variability  in  the  number  and  timing  of  LOA  changes  within  and 
across  participants,  indicating  a  need  to  further  explore  the  utility  of  this  method  to  balance 
workload  between  automation  and  the  operator.  Thus,  another  experiment  was  initiated  to  explore 
whether  results  to  date  reflect  an  attentional  benefit  from  having  the  image  task  autonomy  level 
change  during  trials.  In  other  words,  it  may  be  that  performance  was  improved  in  the  adaptive 
condition,  compared  to  the  static  condition,  simply  because  the  LOA  of  the  image  analysis  task 
changed  during  the  course  of  the  trial  -  not  that  it  changed  in  respect  to  performance. 

In  this  experiment,  a  between-subject  design  was  employed  (Reference  14).  For  one  participant 
group,  the  image  analysis  task  LOA  adapted  according  to  the  performance-based  asymmetrical 
algorithm  parameters  employed  in  the  previous  study.  For  the  other  group,  changes  to  the  task’s 
LOA  were  not  tied  to  task  performance,  but  rather  changed  as  a  function  of  time  elapsed  and  at  the 
frequency  recorded  during  an  earlier  performance-based  adaptive  study.  Data  from  24  participants 
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were  collected  and  the  results  indicated  that  performance  did  not  significantly  differ  between  the 
two  groups.  However,  there  were  significantly  more  autonomy  level  changes  in  the  performance- 
based  adaptive  group.  A  follow-on  study  utilizing  a  yoked-subject  design  is  recommended 
(Reference  14)  to  better  evaluate  the  potential  attention-based  benefit  of  LOA  changes. 

3.1.4  Adaptable  Autonomy  Evaluations 

Another  alternative  to  adaptive  automation  for  re-balancing  human-automation  involvement  in 
task  completion  is  adaptable  automation,  whereby  the  operator  controls  task  LOA.  Using  a  similar 
experimental  protocol  with  the  ALOA  simulation,  a  within-subjects  design  was  conducted  with 
some  trials  employing  a  performance-based  adaptive  control  scheme  (described  earlier)  and  other 
trials  having  an  adaptable  control  scheme  in  which  participants  could  change  the  LOA  of  the 
image  analysis  task  at  any  time  during  the  trial  by  making  selections  on  a  window  of  the  testbed 
(Reference  15).  The  results  showed  that  mean  performance  on  a  different  task,  a  change  detection 
task,  was  slightly  better  with  the  adaptable  condition  compared  to  performance  on  it  under  the 
performance-based  adaptive  condition.  This  result  suggests  that  the  act  of  delegating  LOAs  itself 
may  serve  to  better  keep  the  operator  in-the-loop  and  alert  to  unexpected  stimuli.  It  is  important  to 
note,  though,  that  the  cognitive  overhead  of  delegating  LOAs  in  any  adaptive  system  is  removed 
from  the  decision-making  process.  In  this  respect,  adaptable  automation  can  have  a  performance 
cost,  as  there  is  workload  involved  in  managing  task  LOA.  This  study’s  results  supported  this 
notion,  as  mean  time  to  complete  the  image  analysis  task  was  slightly  longer  with  the  adaptable 
condition  compared  to  the  adaptive. 

Data  on  the  experimental  participants’  personality  were  also  recorded  in  this  experiment 
comparing  adaptable  and  (performance-based)  adaptive  automation.  The  results  showed  a  very 
strong  correlation  between  extraversion  and  autonomy  level  choice  for  the  image  analysis  task: 
highly  extraverted  participants  chose  the  highest  level  of  autonomy,  which  only  required  a 
response  if  they  wanted  to  veto  the  automation’s  recommendation.  In  contrast,  less  extraverted 
participants  chose  a  level  of  automation  (medium)  that  required  the  operator’s  consent  before 
acting  (Reference  15). 

Finally,  while  not  a  focus  of  this  project,  support  was  provided  to  an  Air  Force  Headquarters 
funded  effort  to  explore  a  specific  application  of  an  adaptable  automation  approach  whereby  an 
operator  can  flexibly  change  control  modes  ranging  from  low-levels  of  automation  (manual,  hands 
on  throttle  and  stick)  to  high-level  “plays”  in  which  a  verbal  command  initiates  planning  for  a 
series  of  automated  tasks.  Support  first  involved  conducting  a  literature  search  and  operator 
interviews  to  identify  candidate  applications  of  automation  in  future  UAS  operations  and 
specifically  high-level  plays  that  might  be  useful  (Reference  16).  Next,  design  specifications  were 
determined  for  the  display  and  control  interfaces  that  support  this  multi-level  delegation  control 
concept.  The  design  was  then  instantiated  into  a  prototype  computer  simulation,  with  assistance 
with  respect  to  the  speech  recognition  component  (Reference  17).  The  research  stage  involved  the 
conduct  of  experimental  sessions  with  storyboards  and  the  demonstration  simulation  in  order  to 
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collect  feedback  from  UAS  operators  on  the  usefulness  of  the  “Flexible  Levels  of  Execution- 
Interface  Technologies”  (FLEX-IT)  concept  for  future  UAS  operations,  as  well  as  the  usability  of 
the  display  symbology  and  related  controls  and  procedures.  Ratings  and  comments  from  the 
nineteen  UAS  operators  indicated  that  the  FLEX-IT  approach  for  adaptable  automation  is  indeed 
promising  in  terms  of  providing  intuitive  methods  to  interact  with  automation  as  well  as  seamless 
transition  back  and  forth  between  control  levels  (Reference  18).  This  involves  mechanisms  by 
which  the  pilot  can  make  changes,  navigate  back  and  forth  between  control  levels,  take  control  and 
then  give  it  back  to  the  automation,  all  within  the  shared  understanding  of  “what’s  going  on.”  The 
approach  also  provides  an  ability  to  quickly  tailor  the  automation’s  actions,  as  well  as  an  option  for 
direct  manipulation  inputs  that  assist  in  designating  spatial  locations. 

To  examine  this  multi-level  control  architecture  for  adaptable  automation  further,  the  design 
interface  concept  was  expanded  for  multi-UAS  test  scenarios.  This  also  involved  the  checkout  of 
modifications  made  to  the  concept  demonstrator  as  well  as  the  conduct  of  a  usability  evaluation  in 
which  participants  employed  the  multi-level  control  architecture  during  ninety  minute  sessions. 
Data  included  comments  recorded  with  a  think-aloud  paradigm  and  questionnaire  responses.  The 
results  indicated  that  this  adaptable  pilot-automation  interface  for  multi-UAS  control  is  promising. 
However,  the  findings  reflecting  perspectives  from  both  pilot  and  gamer  participants  indicated  that 
improvements  are  needed  to  enhance  the  interface’s  flexibility  and  usability  (Reference  19). 

A  smaller  scale  experiment  (Reference  20)  was  also  designed  and  conducted  to  quantify  how  long 
it  takes  to  use  an  intermediate  level  of  control  to  specify  maneuvers  to  be  performed  in  the  near 
future.  With  this  novel  control  mode,  the  stick  and  throttle  are  remapped  to  translate  the  operator’s 
inputs  in  azimuth  and  elevation  changes  to  a  directed  flight  path.  This  mode  is  referred  to  as  the 
“noodle,”  as  it  presented  a  flexible  line  segment  resembling  a  bendable  noodle  emerging  from  the 
nose  of  an  UAS  symbol  on  the  map  display.  The  quantitative  data  collected  compliments  the 
qualitative  data  collected  in  earlier  studies  (References  18  and  19)  indicating  this  control  mode  for 
visualizing  and  commanding  near-term  future  path  is  a  definite  candidate  for  future  multi-UAS 
control.  For  example,  the  data  demonstrates  how  the  future  flight  path  can  be  constructed  very 
quickly  (typically  less  than  a  minute).  Comparing  these  data  with  an  estimate  of  time  to 
hypothetically  fly  the  paths  suggests  a  90%  attentional  time  savings  for  typical  operations.  Thus, 
the  ability  to  easily  set  the  complex  flight  path  of  one  vehicle  should  allow  more  time  to  be 
focused  on  other  vehicles,  enhancing  supervisory  control  of  multiple  UASs. 


3.2  Design/Evaluate  Novel  Operator-Automation  Interfaces 

A  second  research  thread  addressed  in  this  project  involved  the  design  and  evaluation  of  novel 
interfaces  for  support  of  multi-UAS  control  and  interaction  with  automation.  Supervisory  control 
of  multiple  UASs  will  require  an  operator  to  be  simultaneously  aware  of  the  status  of  multiple 
vehicles,  including  their  ability  to  meet  any  temporal  constraints  imposed  by  any  cooperative 
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missions.  A  temporal  interface  may  be  useful  for  providing  time-based  information  needed  for 
multi-UAS  supervisory  control  in  addition  to  a  map-based  display  and  other  status  indicators.  The 
vision  for  such  an  interface  is  to  capture  critical  information  in  a  single  window  relevant  to  the 
multiple  vehicles  being  supervised,  along  with  the  missions  they  are  performing  and  also  provide 
control  functionality  such  that  selections  in  the  same  window  will  enable  the  operator  to  make 
inputs  to  automated  sub-systems. 

3.2.1  Evaluation  of  Temporal  Interface  Design  Approaches 

Unfortunately,  there  is  a  dearth  of  research  or  design  guidelines  related  to  temporal  interfaces.  For 
instance,  one  of  many  questions  is  how  the  display  should  be  oriented.  While  a  horizontal 
orientation  is  typically  used  in  Gantt  charts,  a  vertical  orientation  might  be  useful  as  a  scan  from 
left  to  right  could  provide  more  efficient  retrieval  of  the  status  of  all  vehicles  for  a  specific  time 
slice.  This  may  also  help  coordinate  multiple  vehicles  across  time.  To  address  questions  like  these, 
support  was  provided  in  the  functional  design  of  a  part-task  testbed  to  be  used  in  evaluations  of 
candidate  display  formats.  This  testbed  enables  the  evaluation  of  a  variety  of  media  (e.g., 
PowerPoint  slides  and  movie  clips)  and  provides  the  experimenter  with  numerous  options  to 
specify  the  duration  of  the  format  presentations,  the  content  and  viewing  duration  of  questions 
designed  to  measure  ability  to  retrieve  information  from  the  candidate  formats,  the  mechanism  by 
which  the  participants  make  responses,  and  how  the  data  are  recorded.  With  this  new  tool,  several 
issues  related  to  the  design  of  the  temporal  format  were  rapidly  evaluated  without  the  expense  of 
instantiating  the  format  with  software  code. 

In  respect  to  the  question  of  how  the  temporal  format  should  be  oriented,  the  results  of  an 
experiment  conducted  indicated  that  subjects’  performance  (response  time  and  accuracy)  did  not 
significantly  differ  between  the  two  orientation  conditions  (Reference  21).  However,  the  majority 
of  subjective  ratings  were  in  favor  of  the  horizontal  view.  There  were,  however,  certain  tasks  or 
situations  identified  that  would  benefit  from  a  vertical  orientation.  Thus,  research  evaluating 
whether  it  is  beneficial  to  provide  operators  the  option  to  tailor  a  temporal  format’s  orientation  for 
a  particular  mission  or  task  is  needed. 

An  example  of  the  use  of  the  testbed  to  evaluate  dynamic  formats  involved  evaluating  alternatives 
to  portray  the  passage  of  time  (Reference  22).  Specifically,  participants  viewed  two  methods,  each 
at  two  speeds  of  movement.  In  one  method,  the  timeline  moved  from  right-to-left,  in  respect  to  a 
stationary  current-time  vertical  bar.  In  the  other  method,  the  timeline  was  fixed,  and  the  current¬ 
time  vertical  bar  moved  from  left  to  right.  During  the  trials,  participants  watched  the  videos, 
answering  questions  requiring  information  retrieval.  The  results  indicated  that  performance  was 
best  with  the  moving  timeline  condition  at  the  faster  movement  speed,  particularly  on  questions 
requiring  planning  decisions.  Participants’  subjective  data  were  aligned  with  the  performance  data. 
The  results  suggest  that  it  may  be  easier  to  retrieve  information  and  judge  the  temporal  relationship 
of  events  indicated  with  symbology  elements  when  there  is  an  apparent  visual  flow  of  the  timeline 
symbology  in  respect  to  the  fixed,  stationary  frame  of  the  display  window. 
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Use  of  this  testbed  to  evaluate  candidate  formats  and  related  design  issues  provided  useful 
information  for  the  design  of  a  temporal  display  for  multi-UAS  control.  However,  such  results  are 
limited,  for  instance,  because  the  participants’  only  task  was  to  retrieve  information  from  the 
temporal  interface  to  answer  questions.  For  multi- vehicle  control,  operators  will  be  required  to 
switch  attention  across  multiple  tasks,  some  employing  the  temporal  format  and  some  with  other 
interfaces  in  the  control  station  (e.g.,  communications  and  imagery  inspection).  Thus,  a  temporal 
interface  that  is  dynamic  and  used  in  conjunction  with  a  high-fidelity  simulation  of  the  entire  task 
environment  is  required  so  that  the  experimental  participant’s  workload  represents  the  future 
vision  of  multi-vehicle  supervisory  control  applications. 

3.2.2  Prototype  Temporal  Interface  for  Multiple  Vehicles  Supporting  Surveillance  Mission 

Research  focus  next  turned  to  participating  in  the  design  of  a  temporal  display  to  be  instantiated  in 
software  and  integrated  into  a  high-fidelity  multi-UAS  simulation.  Developing  a  dynamic  temporal 
interface  required  several  design  considerations  and  decisions  with  respect  to  the  required  temporal 
data  structures  in  order  for  the  product  to  support  the  changing  realities  inherent  in  a  multiple 
vehicle  control  system.  Choices  in  how  temporal  information  is  stored,  updated,  and  manipulated 
impacts  the  utility  of  the  temporal  interface  and  how  the  operator  will  be  aware  and  interact  with 
temporal  information  (Reference  23).  An  initial  temporal  interface  prototype  was  developed  and 
integrated  into  a  modified  version  of  the  RHCI’s  Vigilant  Spirit  Control  Station  multi-unmanned 
vehicle  simulation  (Figure  3). 


Past  f  Current  Time  Future 


Figure  3.  Prototype  Timeline  Interface  Integrated  into  Vigilant  Spirit  Control  Station 

Horizontal  bars  at  the  top  of  the  window  provided  information  pertaining  to  the  surveillance 
missions  to  be  performed  by  the  unmanned  vehicles.  In  the  lower  portion,  four  color-coded 
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horizontal  bars  were  presented,  one  for  each  of  four  vehicles  being  supervised,  along  with  overlaid 
symbology  providing  vehicle  specific  information.  Interactions  with  the  symbology  could  be  made 
to  manage  vehicle  systems  (e.g.,  return  a  vehicle  to  base  for  refueling)  and  manage  missions  (e.g., 
pulling  bar  representing  new  mission  down  to  a  specific  vehicle’s  bar  to  make  the  corresponding 
mission-vehicle  assignment). 

A  human-in-the-loop  simulation  evaluation  was  conducted  to  determine  if  the  addition  of  this 
prototype  temporal  display  improves  supervisory  control  performance.  Participants  completed 
experimental  trials  with  and  without  the  temporal  interface.  For  trials  without  the  temporal 
interface,  participants  completed  steps  via  multiple  windows,  mainly  via  interactions  with 
symbology  overlaid  on  the  map.  Participants  were  required  to  complete  multiple  types  of  tasks 
representative  of  the  tasks  envisioned  for  multi-vehicle  supervisory  control.  For  all  tasks,  response 
time  and  accuracy  measures  were  recorded.  The  results  showed  that  performance  was  better  for  all 
task  types  when  the  temporal  interface  was  present  (results  significant  for  eight  of  twelve  task 
types).  Not  only  was  performance  better  for  tasks  involving  interactions  with  the  temporal 
interface,  but  also  for  secondary  tasks  involving  peripheral  windows.  This  finding  suggests  that  the 
temporal  interface  helped  reduce  overall  workload  and  improve  situation  awareness,  such  that 
more  attention  could  be  devoted  to  other  tasks.  Subjective  data  were  aligned  with  the  performance 
data.  Documentation  of  the  results  is  underway. 

3.2.3  Prototype  Temporal  Interface  for  Multiple  Vehicles  Coordinating  on  Time-sensitive 
Target  Prosecution 

While  both  the  performance  and  subjective  results  were  favorable  in  the  evaluation  of  the  temporal 
interface  described  above,  the  surveillance  missions  represented  in  the  experimental  trials  were  not 
especially  complex  in  that  each  was  performed  by  a  single  vehicle  and  there  were  few  temporal 
constraints.  It  was  decided  to  next  design  a  temporal  interface  that  could  support  multiple  vehicles 
that  coordinated  actions  with  respect  to  time-critical  targeting  tasks.  In  this  regard,  a  supporting 
use-case  scenario  was  identified,  as  well  as  the  design  of  display  elements  and  control  functionality 
for  the  novel  interface  (Figure  4).  It  was  determined  that  a  task-centric  temporal  format,  rather  than 
a  vehicle-centric  format  (Figure  3)  was  more  appropriate  for  the  time-critical  tasks  that  involved 
multiple  steps  for  each  target,  including  both  sensor-equipped  vehicles  and  weapon-equipped 
vehicles.  Each  horizontal  line  presented  information  related  to  one  target  that  required  imaging  and 
prosecution  before  a  deadline.  To  support  the  selection  of  vehicle  resources  and  appropriate 
allocation  of  attention  across  the  tasks,  several  different  levels  of  automated  aids  were  also 
instantiated.  In  one  lower  level  of  automation,  deadlines  for  the  overall  target  task,  as  well  as  when 
the  image  of  the  target  would  expire,  were  indicated  with  symbology.  In  a  higher  level  of 
automation,  additional  symbology  (including  colored  zones  overlaying  the  timeline)  was  presented 
to  show  recommended  deadlines  for  subtasks  in  the  target  prosecution  cycle  as  well.  A  third  level 
provided  an  additional  decision  aid,  highlighting  which  of  multiple  vehicles  the  automation 
recommended  that  the  operator  should  select  for  the  current  targeting  task.  The  design  also 
featured  novel  “glyphs”  in  which  individual  dimensions  (variables)  are  mapped,  in  real-time,  to 
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attributes  of  a  graphical  shape  or  symbol.  These  dynamic  information-packed  symbols  are 
designed  to  conserve  display  space  and  reduce  the  operator’s  need  to  visually  scan  and  integrate 
information  across  the  control  station.  For  this  prototype,  different  glyphs  are  used  to  denote 
progress  on  the  time-sensitive  target  tasking  as  well  as  the  status  of  the  different  UAS  types  (e.g., 
sensor  and  weapon).  Research  examining  this  new  prototype  was  initiated,  but  not  completed  in 
the  reportable  timeframe  of  this  report. 


Glyphs  showing 
task  and  UAS 


Vehicle 

Selection  Panel 


status 


Figure  4.  Prototype  Timeline  Interface  Windows  Integrated  into  Vigilant  Spirit  Control  Station  to 
Support  Multiple  Vehicles  Coordinated  on  Time-sensitive  Target  Prosecution 


3.2.4  Operator-Automation  Interface  Design  Inspired  by  Finite  State  Automata  Graphs 

Besides  the  efforts  described  above  that  focused  on  a  timeline  interface  that  has  both  display  and 
control  functionality,  another  design/development  effort  supported  used  a  different  approach  for 
providing  the  operator  with  a  display  that  supports  “at-a-glance”  understanding  of  the  current  state 
of  all  vehicles  and  missions  being  managed.  This  approach,  termed  “L-PRISM”  (Layered  Pattern 
Recognizable  Interface  for  State  Machines),  is  based  on  a  Finite  State  Automata  Graphical 
technique  (see  Figure  5).  The  larger  display  elements  in  the  display  window  represent  major 
“operational  states” — segments  of  a  mission  being  performed  by  multiple  LFASs,  along  with 
automated  sensing  ground  sensors.  Symbology  (icons  and/or  alphanumeric  text)  within  each  major 
display  element  indicates  the  vehicles  currently  performing  tasks  related  to  that  mission  “state.” 
Thus,  with  a  quick  glance  across  the  display  elements,  the  operator  can  identify  the  mission  states 
and  vehicle/sensor  taskings  from  the  template.  Moreover,  the  temporal  relationships  (past,  present, 
future)  between  the  various  states  can  be  depicted,  as  well  as  cues  of  task/state  transitions.  This  use 
of  pattern  recognition  to  depict  the  relationships  of  the  different  entities  may  enhance  operator’s 
situation  awareness,  especially  when  highly  automated  collaborating  systems  can  have 
autonomous  decision  making  capabilities  within  a  decentralized  control  architecture.  For  such 
applications,  it  is  even  more  critical  that  the  functionality  of  the  automation  is  transparent  as  the 
operator  may  only  have  limited  visibility  and  control  of  certain  entities.  The  L-PRISM  design  also 
provides  means  for  the  operator  to  manipulate  multi-vehicle  goals,  tasks,  and  constraints  with  less 
point  and  click  menu  navigation. 
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Figure  5.  Illustration  of  L-PRISM  prototype: 

Layered-Pattern  Recognizable  Interfaces  for  State  Machines 

This  includes  a  pictographic  temporal  tool  below  the  larger  window  that  provides  a  mechanism  for 
the  operator  to  store,  recall,  and  view  imagery  and  other  information  packets  transmitted  from  air 
and  ground  entities.  The  effort  to  date  has  involved  design  of  the  display  interface  and  instantiation 
of  the  interface  in  Vigilant  Spirit  Control  Station  software  (Reference  24).  Support  has  also  been 
provided  in  preparations  underway  to  evaluate  the  interface  in  simulations  and  actual  live  tests. 
This  effort  is  part  of  a  larger  Collaborative  Systems  Control  Strategic  Technology  Thrust  multi- 
AFRL  Technical  Directorate  Initiative  called  “Value  of  Information  in  Collaborative  Systems.” 


3.3  Other  Projects 

One  effort  involved  personnel  with  OR  Concepts  Applied,  the  firm  that  designed  the  initial  version 
of  the  ALOA  simulation  via  an  earlier  Small  Business  Innovative  Research  effort.  Several  features 
of  the  test  simulation  were  improved  and  expanded  to  increase  the  functionality  and  usability  of 
the  simulation  to  explore  the  balance  of  autonomy  with  operator  involvement. 

Personnel  from  Alion  Science  &  Technology  conducted  another  effort.  Several  gaming 
technologies  were  evaluated  to  identify  display  and  control  concepts  that  might  be  useful  for 
application  in  the  control  of  UASs.  Gaming  technologies  evaluated  ranged  from  massively 
multiplayer  online  role-playing  games  to  arcade-style  games.  While  this  assessment  identified 
potential  research  areas,  the  similarities  among  many  games,  and  the  shortage  of  games  that 
address  some  of  the  key  problems  with  unmanned  systems  (mainly  switching  between  the 
supervisory  role  and  the  direct  control  role),  made  this  effort  less  fruitful  than  originally 
anticipated.  A  probable  reason  relates  to  the  differing  primary  goals  of  games  and  real  UAS 
control.  Games  are  designed  to  maximize  the  user’s  sense  of  engagement,  rather  than  providing  the 
most  efficient  means  for  accomplishing  a  given  mission.  Hence,  enhancing  mission  performance, 
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while  reducing  operator  workload  is  not  a  goal  in  games.  This  and  other  fundamental  differences 
complicate  translating  successful  concepts  from  one  domain  to  another.  There  were  two 
publications  documenting  this  work:  a  technical  report  and  a  conference  paper  (References  25  and 
26,  respectively). 


4.  SUMMARY 

This  report  provides  a  summary  of  research  conducted  by  the  Supervisory  Control  and  Cognition 
Branch  (RHCI)  of  the  711*  Human  Performance  Wing  of  the  Air  Force  Research  Laboratory  that 
examines  the  balance  of  autonomy  with  operator  involvement.  Efforts  during  this  reporting  period 
also  involved  the  design  and  evaluation  of  novel  operator-automation  interfaces.  Continuation  of 
this  research  will  be  summarized  in  the  final  report  for  Task  Order  49,  performed  in  support  of  the 
Warfighter  Interface  Research  &  Technology  Operations  (WIRTO)  contract  (FA8650-08-D-6801). 
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6.  LIST  OF  ACRONYMS  AND  ABBREVIATIONS 


Acronym 

Definition 

AFRL 

Air  Force  Research  Laboratory 

ALOA 

Adaptive  Levels  of  Autonomy 

BATC 

Ball  Aerospace  &  Technologies  Corporation 

FLEX-IT 

Flexible  Levels  of  Execution  -  Interface  Technologies 

LOA 

Level  of  Automation 

L-PRISM 

Layered  Pattern  Recognizable  Interface  for  State  Machines 

RHCI 

Supervisory  Control  and  Cognition  Branch,  Warfighter  Interface  Division,  Air  Force 
Research  Laboratory,  Human  Effective  Directorate,  71 1*  Human  Performance  Wing 

UAV 

Unmanned  air  vehicle 

UAS 

Unmanned  air  vehicle  system 

WIRTO 

Warfighter  Interface  Research  &  Technology  Operations 
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